Skip to content

feat(library): import curated collections + upload your own documents#20

Merged
FzzySwtr merged 2 commits into
claude/app-models-env-issue-15db45from
claude/content-library-import
Jul 17, 2026
Merged

feat(library): import curated collections + upload your own documents#20
FzzySwtr merged 2 commits into
claude/app-models-env-issue-15db45from
claude/content-library-import

Conversation

@FzzySwtr

Copy link
Copy Markdown
Contributor

Stacked on #19 (self-healing model loading). Base is that branch so this PR shows only the import diff — merge #19 first, then this retargets to main cleanly.

Why

There was no user-facing way to grow the field library — content could only be added via CLI/compose. This adds a way to import libraries from one curated place (public-domain, categorized) and to upload your own documents, both in-app and via CLI. New files land in the sources volume where ingestion already auto-indexes them — no rebuild, no restart.

What

Catalog — sources.yaml is now the single source of truth

  • A top-level collections: block groups the 32 public-domain/CC docs into 8 named bundles (Core Survival, Austere Medicine, Food & Water, Power & Engineering, Comms & Radio, Education, Civics & Culture, Software); every item carries a collection: tag alongside its category.
  • License values containing # are quoted (so they're not parsed as YAML comments).

CLI — fetch-source-data.sh

  • --collection <id>, --list-collections; --validate now checks each item references a declared collection.

Server — src/catalog.h (new) + src/server.cpp

  • GET /api/catalog — collections + per-item installed state.
  • POST /api/import { "collection": id } — validates the id against the catalog (closed set) and as a strict slug; single-flights via an atomic flag; downloads by posix_spawn-ing the bundled fetcher with an explicit argv (never a shell). Only the fixed, image-baked manifest URLs are ever contacted → no arbitrary-URL / SSRF surface.
  • POST /api/upload — multipart PDF/TXT; filename reduced to a safe basename, category constrained to NNN_Name, %PDF magic + size checks, a free-space guard, and an atomic link() finalise that fails closed (EEXIST → 409) so a check-then-rename race can't clobber a file.
  • /query and /api/import reject bodies over MAX_REQUEST_BODY; global body cap bounded (MAX_UPLOAD_BYTES = 32 MB).

UI — public/

  • A "+ Add" action on the Field Library panel opens a modal: collections with install progress + a PDF/TXT upload drop zone. CSP-safe (no inline handlers), design-canon clean.

Air-gapped: JIC_DISABLE_NETWORK_IMPORT disables network import; uploading local files stays available.

Verification

  • End-to-end (17/17): catalog lists 8 collections; a real collection (comms-and-radio) downloads and indexes; an uploaded .txt gets indexed; installed-state flips.
  • Security (17/17 exercised + adversarial review → SHIP): ../../../pwned.txt sanitized to stay inside its category; injection-y import id → 404 (no shell); traversal category → 400; fake-PDF → 400; single-flight import → 409; disk/DoS guarded.
  • Degraded test-server.sh 21/21 · test-config.sh 38/38 · lint-canon · unit tests · clean image build.

Reviewer notes

  • Import shells out via posix_spawn with a validated, closed-set collection id and an explicit argv — no /bin/sh, no client-supplied URL. Works under the read_only/cap_drop: ALL hardening (curl + the fetcher + sources.yaml are image-baked; writes go to the writable sources volume).
  • The 32 MB global body cap is an accepted, documented tradeoff on this single-user offline appliance (cpp-httplib's payload limit is process-global). Non-blocking follow-up: move the upload to a streaming multipart ContentReader to drop even the transient buffer.

🤖 Generated with Claude Code

Adds a way for users to grow the field library at runtime, from one curated
place, without a rebuild or restart. Content lands in the sources volume where
the ingestion service already auto-indexes it.

Catalog (sources.yaml):
- Becomes the single source of truth: a top-level `collections:` block groups
  the 32 public-domain / CC documents into 8 named bundles (Core Survival,
  Austere Medicine, Food & Water, Power & Engineering, Comms & Radio,
  Education, Civics & Culture, Software), and every item carries a
  `collection:` tag alongside its category. License values containing '#'
  are quoted so they aren't parsed as YAML comments.

CLI (helper-scripts/fetch-source-data.sh):
- Parses the new field; adds `--collection <id>` and `--list-collections`;
  --validate now checks each item references a declared collection.

Server (src/catalog.h new, src/server.cpp):
- GET  /api/catalog  — collections + per-item "installed" state.
- POST /api/import   — { "collection": id }; validates the id against the
  catalog (a closed set) AND as a strict slug, single-flights via an atomic
  flag, and downloads by posix_spawn-ing the bundled fetcher with an explicit
  argv (never a shell). Only the fixed, image-baked manifest URLs are ever
  contacted — no arbitrary-URL / SSRF surface.
- POST /api/upload   — multipart PDF/TXT; filename reduced to a safe basename,
  category constrained to NNN_Name, %PDF magic + size checks, a free-space
  guard, and an atomic link()-based finalise that fails closed (EEXIST → 409)
  so a check-then-rename race can't clobber an existing file.
- Small JSON endpoints (/query, /api/import) reject bodies over
  MAX_REQUEST_BODY; the global body cap is bounded (MAX_UPLOAD_BYTES = 32 MB).

UI (public/): a "+ Add" action on the Field Library panel opens a modal that
lists the collections (with install progress) and a PDF/TXT upload drop zone;
CSP-safe (no inline handlers), design-canon clean.

Air-gapped deployments can disable network import with JIC_DISABLE_NETWORK_IMPORT
(uploading local files stays available).

Verified: end-to-end (catalog lists 8 collections; a real collection downloads
and indexes; an uploaded doc is indexed) and security (path-traversal, command
injection, SSRF, single-flight, disk/DoS) — 17/17; plus degraded test-server.sh
21/21, test-config.sh 38/38, lint-canon, unit tests, clean image build. The
change passed an adversarial security review with a SHIP verdict.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Copilot AI review requested due to automatic review settings July 17, 2026 05:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a user-facing “grow the library” workflow to JustInCase by introducing a curated content catalog (importable by collection) plus a direct PDF/TXT upload path that writes into the existing sources volume for ingestion to auto-index.

Changes:

  • Add server endpoints for catalog listing, collection import (via posix_spawn), and multipart document upload with validation/guards.
  • Define curated “collections” in sources.yaml and extend the fetch helper to filter/list by collection and validate collection membership.
  • Add a UI modal (+ CSS/JS wiring) to install curated collections and upload local documents.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/server.cpp Adds /api/catalog, /api/import, /api/upload, and payload-size gating; implements import worker spawning and upload validation/storage logic.
src/config.h Introduces MAX_UPLOAD_BYTES and config/env helpers for catalog path, fetcher path, and network-import enablement.
src/catalog.h New minimal manifest parser for collections + items used by /api/catalog and /api/import validation.
sources.yaml Adds top-level collections: and tags each source item with a collection:.
public/index.html Adds “+ Add” action and modal markup for curated import + upload UI.
public/app.js Implements modal open/close, catalog rendering, import triggering + polling, and upload flow/drag-drop.
public/style.css Styles the new “add content” modal and catalog/upload UI elements.
helper-scripts/fetch-source-data.sh Adds --collection, --list-collections, and validation that each item references a declared collection.

Comment thread src/server.cpp
Comment thread src/server.cpp Outdated
Comment thread src/server.cpp Outdated
Comment thread public/app.js
…ling

Four Copilot review findings on the content-import feature:
- /api/upload: reject a missing "file" form field with a clean 400 (has_file
  check) instead of risking a throw/500 on a malformed multipart body.
- /api/upload: validate the extension case-insensitively (e.g. FOO.PDF), and
  run the %PDF magic-byte check for any-case .pdf — matching ingestion, which
  lowercases extensions.
- /api/upload: fail CLOSED on the free-space guard — if create_directories or
  fs::space errors, reject (500/507) rather than writing without the disk-full
  protection.
- app.js: clear the import progress poll when the modal closes (no more
  background /api/catalog fetches every 3s), and resume it on reopen if an
  import is still running.

Verified: missing-file→400, uppercase .PDF/.TXT accepted, uppercase .pdf
without %PDF magic still →400; clean image build; app.js node --check + lint OK.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
@FzzySwtr
FzzySwtr merged commit e592016 into claude/app-models-env-issue-15db45 Jul 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants